Skip to content

fix(writer): a numeric column keeps its dict layout past 2,048 distinct values (#649) - #682

Merged
dfa1 merged 1 commit into
mainfrom
fix/649-numeric-dict-runs
Nov 7, 2026
Merged

dfa1 merged 1 commit into
mainfrom
fix/649-numeric-dict-runs

Conversation

@dfa1

@dfa1 dfa1 commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Fixes #649.

  • Numeric columns take Rust's dictionary-run flow: admitted by the probe (cascade on the first 8192 rows), dictionary closed at 65,535 entries / 1 MiB, next one started; several dictionaries are a chunked layout of dict layouts. The 2,048 cap and the demotion are gone.
  • cascading(0) has no cascade to probe, so the sample's distinct count decides (under half, more than one).
  • Interop test vs vortex-jni: same dict-layout count for i64 pools of 100–150000 values, and for a column whose first 8192 rows are low-cardinality followed by unique values.

Builds on the text flow merged in #679. Decimal columns (#662) are separate.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

…ct values, one dictionary after another (#649)

Rust's DictStrategy admits a column when its probe compressor, the cascade, turns the first 8192
rows into a dict, and closes the open dictionary at 65,535 entries or 1 MiB (the primitive builder's
max_len and max_bytes / byte width) to start the next. Java admitted a numeric column only with at
most 2,048 distinct values, under half distinct, in one dictionary demoted to per-chunk encoding
past the cap, so hash-like columns (ClickBench RefererHash, URLHash) were written plain, twice the
size.

Numeric columns now take the same prefix-consuming ingest as text: the first entry is always
admitted, the row that needs an entry that no longer fits (a value or the null entry) ends the
dictionary, and the writer chunks the dict layouts. Admission is Rust's probe for every nullable
or non-null column of a type the layout reads; with cascading(0) there is no cascade, so the
sample's distinct count decides. The interop test compares the dict-layout count with vortex-jni
for pools of 100 to 150000 i64 values.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 merged commit 6d6857c into main Nov 7, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/649-numeric-dict-runs branch November 18, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries

1 participant